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Abstract — Teleoperation experiments at JPL have 
shown that advanced features in a telerobotic system are 
a necessary condition for good results, but that they are 
not sufficient to assure consistent good performance by 
the operators. Two or three operators are normally used 
during training and experiments to maintain the desired 
performance. An alternative to this multi-operator con- 
trol station is a man-machine interface embedding com- 
puter programs that can perform some of the operators 
functions. 

In this paper we present our first experiments with 
these concepts, in which we focused on the areas of real 
time task monitoring and interactive path planning. In 
the first case, when performing a known task, the oper- 
ator has an automatic aid for setting control parameters 
and camera views. In the second case, an interactive 
path planner will rank different path alternatives so that 
the operator will make the correct control decision. The 
monitoring function has been implemented with a neural 
network doing the real-time task segmentation. The in- 
teractive path planner has been implemented for redun- 
dant manipulators to specify arm configurations across 
the desired path and satisfy geometric, task and perfor- 
mance constraints. 

I Introduction 

Advanced teleoperation systems are characterized by 
computerized features aimed at reducing the operator’s 
effort and enhancing his concentration during tasks. 
Typical features are force reflecting control joysticks, 
mixing of video images with computer graphics and ad- 
vanced control modes such as sensor or model-referenced 
control. These features achieve a level of transparency 
between operator and task that assures good awareness 
of the remote task status. 


The teleoperation experiments carried out in the 
Advanced Teleoperation (ATOP) laboratory of the Jet 
Propulsion Laboratory (JPL) have shown that these fear 
tures are necessary for good results, but not sufficient 
for consistent performance. The control of the many 
complex features of advanced teleoperators often results 
in excessive mental load and fatigue even during sim- 
ple realistic experiments. To avoid this potential prob- 
lem, two, often three, operators are used to manage the 
ATOP system during training and experiments. A pri- 
mary operator is in charge of the manipulation and the 
others carry out support functions or serve as trainers. 
The multioperator approach is convenient when there are 
no constraints on the manpower available at the control 
station, but, when its volume is constrained, the man- 
machine interface should be controllable by a single per- 
son without compromising performance and task comple- 
tion time. Thus the need for automatize some of these 
functions. 

Most of the activities of the supporting operators are 
quite simple and yet very time consuming and could, 
in principle, be replaced by computer programs embed- 
ded in the man-machine interface. Some of these op- 
erations include setting the manipulator control gains, 
moving cameras and lights and operating the data ac- 
quisition functions. All the values in these functions 
are predetermined and the only free variable is the time 
at which those functions have to be done. Other fea- 
tures that should be performed under computer supervi- 
sion because of their computational requirement, axe task 
and path planning. A man-machine interface embedding 
these tools would cooperate with the primary operator 
by providing direct suggestions, presenting command al- 
ternatives and monitoring performance fluctuations. 

To test the feasibility of automatic tools of this type, 
we have implemented two prototypes of an automatic 
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Figure 1: The JPL Dual Arm Advanced Teleoperator 


monitoring program and of a redundant manipulator 
planner. The first tool is based on a neural-network for 
recognition of task phases and real time task measure- 
ment and it will permit the flexible automation of some 
task activities and the generation of feedback messages 
to the operator. In this way, a cooperative environment 
can be set, in which the telerobot and the operator share 
duties and monitoring functions of a teleoperated task. 
This approach would extend the paradigm of supervised 
control to the case in which the telerobot monitors and 
supervises the operator actions [4]. 

The second tool that we have also developed is an 
interactive path planner for redundant robots that al- 
lows the operator to specify arm configurations across the 
desired path satisfying geometric, task and performance 
constraints. Redundant manipulators are designed to en- 
hance dexterity and robustness in task execution because 
they are equipped with more degrees of freedom (dof) 
than the minimum required by the task space. These 
manipulators allow the operator to carry out the pri- 
mary manipulative task and, at the same time, to satisfy 
additional constraints such as avoidance of singularities 
and joint limits, maximization of manipulability indices, 
satisfaction of impedance characteristics, collision avoid- 
ance and fault tolerance. These constraints are satisfied 


by using the redundant dof’s of the manipulator, while 
maintaining the position and orientation required by the 
task. These motions of the manipulators are called self 
motions and the space in which they are executed is the 
manipulator null space. 


When redundant manipulators are used in teleoper- 
ation, there are additional technical issues that relate to 
the operator’s role in resolving the redundancy. Informa- 
tion must be provided to the operator to understand the 
effects of self motions in A meaningful and natural form. 


Next section gives a brief description of the elements 
of the ATOP system and of its operator interface. Our 
initial results in developing a task model that can be 
used for interface/operator interaction axe presented in 
section III. Section IV presents a brief summary of the 
control problems of redundant manipulators, of the spe- 
cific issues of redundant teleoperation and of our param- 
eterization approach [7]. The conclusion summarizes the 
paper and presents our current research and development 
directions. 
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Figure 2: The JPL ATOP Operator Interface 


II The Advanced Teleoperation 
System 

The Advanced Teleoperation Laboratory (ATOP) is 
physically divided into two parts: the remote site and 
the local site. The main features of the remote site are: 

1. A Camera Positioning System consisting of a steel 
frame capable of supporting five movable TV cam- 
eras. Currently, the frame supports three cameras 
mounted on pan/tilt platforms attached to com- 
puter controlled beams moving on the frame sides 
and ceiling. A monoscopic camera is mounted on 
the ceiling and one on a side beam, while a stereo 
camera is mounted on the other side beam. 

2. A Dual Arm System consisting of two AAI 8 degree- 
of-freedom redundant manipulators, two Model C 
JPL Smart Hand and two sets of Universal Mo- 
tor Controllers (UMC) systems (fig. 1). The UMC 
electronics performs the joint servo, the inverse and 
direct kinematics of the redundant manipulators, 
and handles the communication with the smart 
hands and with the local site via fiber optic se- 
rial lines. Several advanced control features are 
available in the dual arm system, such as contour 


following , shared compliance and harmonic motion 
generator [1], [8]. 

3. Controlled Light Sources to create viewing condi- 
tions similar to those found in a space environment. 

The local site of the ATOP system consists of the 
control room with the operator interface (fig. 2). This 
interface has evolved following the development and the 
integration of new technologies into the teleoperation sys- 
tem and is organized in three main subsystems: 

1. A Data Interface consisting of: 

(a) Parameter acquisition interface to enter robot 
configuration and control parameters. 

(b) Data acquisition interface to handle the com- 
munication with the smart hand and process 
incoming force and torque data. 

(c) Programming, debugging and set up interface 
for the development, monitoring and setup of 
the control programs for the manipulators. 

(d) Sensors interface to visualize in real time force 
and torque data collected by the manipulators 
sensors. 
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2. A Video Interface composed of: 

(a) Video Monitors , displaying two monoscopic 
and one stereo views of the remote site, 

(b) Computer Graphics Simulation , equipped 
with predictive display, for time delay com- 
pensation and operator training. 

3. A Manual Interface consisting of: 

(a) Force Reflecting Hand Controllers (FRHC), 
driven by a pair of UMC’s to command the 
movements of the robots and to feed back 
force information to the operator. 

(b) Camera Gantry Controls , to move the beams 
holding the cameras to the desired viewing po- 
sition. 

(c) Camera Controls to adjust pan, tilt, focus, 
zoom and iris of the cameras. 

(d) Foot Pedals for the power tools in the remote 
site. 

Ill Automatic Supervision 

An efficient single-person teleoperation interface must 
perform some of the functions assigned to the opera- 
tors of a multi-operator interface. To experiment with 
this idea, two different functions are under study: the 
first one is the automatic display of camera and control 
menus during a task, and the second is the generation of 
feedback messages to the operator. In the first case, the 
interface must automatically show the operator the ap- 
propriate command menus for the specific task’s phase. 
In the second case, the interface must provide the oper- 
ator with performance feedback messages derived from a 
stored model of the task execution. In both cases, a key 
element of these advanced tools is a program that can fol- 
low the development of a teleoperated task by segmenting 
the sensory data stream into appropriate phases. 

A task segmentation program of this type has been 
implemented by means of a Neural Network architecture 
[2] and it is able to identify the segments of a peg-in-hole 
task. With this architecture, the temporal sequence of 
sensory data generated by the wrist sensor on the ma- 
nipulators are turned into spatial patterns and a window 
of sensor observations is associated to the current task 
phase. 

This problem is referred in the literature as that of 
learning time sequences and has been approached primar- 
ily with two architectures: Time-Delay and Partially Re- 
current networks [5]. Partially Recurrent Networks bet- 
ter represent the temporal evolution of the task since they 


include in the input layer a set of nodes connected to the 
output units, to create a context memory. These units 
represents the task phase already executed - the previous 
state. A set of fixed weight connections have been es- 
tablished among the output and context layer units (see 
figure 3). 



Figure 3: Partially Recurrent Neural Network 


Training is carried out to associate the previous state 
and the window of sensor signals to the current state: 

0-1 (t - At), ..,a n (t - A t),x(t -(l - 1)A t),..,x(t) 

where (<7i(<), . . . ,<7 n (0) ls the state corresponding to the 
x- force sensor signal z(2). The window length (/ value) 
is a critical design factor. 

The Partially Recurrent network gave good results 
both in training and in simulation and it has been inter- 
faced to the ATOP telemanipulator system for real-time 
tests. The ATOP configuration is significantly different 
from the one used for the training data and therefore 
these tests also verified the robustness of the approach 
to hardware variations. 

Figure 4 shows as a dotted line the output of the real- 
time segmentation performed by the network during an 
actual peg-in-hole task. The solid line represents the cor- 
respondence between task’s phases and data. The time 
response of the network is evident in the lag between ho- 
mologous transitions between the solid and the dotted 
lines and it depends primarily on the computer used to 
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Figure 4: Segmentation in the real-time experiment 

collect data and perform the network calculations. Sev- 
eral experiments have been carried out and the results 
have been quite encouraging with a percentage of cor- 
rect segmentations approximately equal to 65%. Figure 
4 refers to the a typical Peg-in-Hole phases’ sequence [3]. 
Approximately at time 29 sec the network misclassifies 
the end of the extract phase (index n. 8) that should have 
occurred at time 34 sec. The network showed also unex- 
pected capabilities to recover after misclassification and 
it was also able to follow tasks whose phases’ sequence 
varied from the training one. During peg extraction, for 
example, if the operator decided to regrasp the peg, the 
network was sometimes able to make the transition from 
extraction to insertion and again to extraction . These 
tests were quite encouraging and this type of architec- 
ture can provide a building block for the automatic tools 
of an advanced teleoperation interface. 

IV Cooperative Redundancy 
Management 

For redundant manipulators it is not possible to formu- 
late a closed form solution of the inverse kinematic prob- 
lem and many control algorithms use a mixture of ve- 
locity based control and optimization procedures. The 
inverse Jacobian is used to compute the velocity trans- 
formations and a local optimization of some evaluation 
criteria is used to determine the values of the redundant 
joints in the arm null space [9], [6]. The above velocity- 
based kinematic control has several drawbacks, including 
computational cost, numerical instability when the Jaco- 


bian inversion is performed near singularities and the lack 
of global optimization of the arm configurations. 

To eliminate these problems, a parameterization ap- 
proach has been proposed for the control of redundant 
manipulators [7]. The redundant joints are considered as 
parameters of a non-redundant manipulator for which we 
can obtain a closed form solution of the inverse kinemat- 
ics. This approach transforms the problem of controlling 
a redundant arm into that of selecting optimal values for 
the configuration parameters, i.e. the redundant joints, 
along the required trajectory. 

This method has computational and numerical ad- 
vantages over velocity-based kinematic controls and it 
provides the necessary support for implementing cooper- 
ative redundancy management in teleoperation. In fact, 
the operator can be shown a display of the parameter 
space in which a surface representing a performance cri- 
terion is visualized. Every point on the surface is associ- 
ated with values of the redundant joints and the operator 
can immediately determine the best configuration. The 
values of the redundant joints are the parameters that are 
used to compute the inverse kinematics of the associated 
non redundant manipulator. 

More specifically, a null space manifold is formed in 
the parameter space with the consideration of individual 
joint limits, and characterized by an artificial potential 
field representing the performance associated with the 
individual null space joint configurations. A null space 
manifold can be easily formed by varying the parameter 
values within their limits, and by checking the availability 
of a solution through the position-based kinematic solu- 
tion. The manifold can be incrementally updated and 
animated along the given task trajectories. The artificial 
potential function is then formed over the null space man- 
ifold based on a combination of several desired attributes 
such as proximity to joint limits, proximity to singulari- 
ties, and measures of kinematic and dynamic manipular 
bility. The potential function represents the performance 
criterion for the selection of the joint configuration. It 
can be used either automatically by the system, follow- 
ing a local gradient search, or manually by the operator. 
In this case, globally optimal joint configurations may be 
selected by extrapolating the variations of performance 
associated with the selected parameter values, or by an- 
alyzing the variation of the potential function at succes- 
sive task points along the given task trajectory. In short, 
the parameterization method allows the visualization of 
manipulator internal performance through the display of 
potential functions in the parameter space. This provides 
a medium for an interactive and cooperative interface for 
redundant telemanipulator planning, through which the 
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operator can decide whether, when and how to reconfig- 
ure the arm for optimal task execution. 



Figure 5: Criterion functions on the 2-D null space 

The AAI arms used in the ATOP laboratory are de- 
composable arms and particularly suited for the proposed 
parameterization method that can be applied to each one 
of the two subarms in which the manipulators can be 
partitioned. In this case, the parameterization method 
is very simple because a closed form of the parameterized 
inverse kinematic solution can be readily obtained from 
each subarm and the null space manifold in the parame- 
ter space can be easily described. 

The AAI arm is an 8 dof manipulator, where the 4 
lower revolute joints are configured as a cascaded spheri- 
cal arm for positioning and the 4 upper revolute joints are 
configured as a Z-X-Y-Z wrist for orientation. The AAI 
arm has no link offsets and since the 4 wrist axes intersect 
at one point, the arm is decomposable. This manipula- 
tor permits an easy kinematic control because of its zero 
offsets and orthogonal mechanical structure and it also 


achieves a high dexterity and singularity avoidance. 

The proposed parameterization method has been suc- 
cessfully applied to the derivation of the closed form, pa- 
rameterized inverse kinematic solution of AAI manipula- 
tors. The derivation was simplified by taking advantage 
of this arm decomposability: the inverse position trans- 
formation uses two joints as parameters, one in the lower 
arm (first 4 joints) and the other in the upper arm (last 
4 joints). A potential function over the solution manifold 
is then mapped onto the parameter space for a given task 
point, as illustrated in fig. 5. 

In our current implementation, the operator is pre- 
sented with a 3-D graphic display of the criterion func- 
tions plotted over the null space of the robot for five 
different end effector positions and orientations. Corre- 
sponding to each end effector position and orientation, a 
surface is plotted showing the value of the criterion func- 
tion over the two dimensional null space. The operator 
can specify the weights for each criterion function for the 
different surfaces. The display seen by the operator is 
shown on Fig. 5. 

The input to the program is with a graphic interface 
and it consists values of the redundant joints with which 
to start or to tune the computation of the optimal tra- 
jectory. This approach to redundant path planner offers 
a number of advantages to the operator for constructing 
feasible paths during a task. In particular, the operator 
has complete control in selecting the robot configuration 
and the process is carried out in real-time since the tool 
provides immediate visual feedback on the quality of the 
selected parameters. 

V Conclusions 

In this paper we have described our current efforts to 
improve the communication between the operator of a 
telerobotic system and the system itself. The long term 
objective is to establish a duality between the human 
supervision of a telerobot, as in the supervisory control 
paradigm, and the automatic support to the user of an 
advanced teleoperator. Two main areas of research have 
been described. The first one aims at developing tech- 
niques for supporting the operator during task execution. 
The type of support envisioned within the interface will 
consist of task segment identification for the automatic 
setting of system parameters and for performance moni- 
toring. The second research effort is concerned with the 
development of an interactive path planner that cooper- 
ates with the operator in the selection of the best path 
and configurations of a redundat telemanipulator. These 
automatic features will be an asset in all phases of tele- 
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Figure 6: Operator Interaction with the Planner 


operation and will be essential in supporting real teleop- 
erated tasks. 
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